This application relates to malware programs and malware protection programs.
Malware programs identify vulnerabilities in binary files, and then execute malicious code exploiting the vulnerabilities. A vulnerability targeted by malware programs, for example, is the assignment of a value to a variable without performing any bounds checking on the value. Because some programming languages (e.g., C and C++) do not perform any automatic bounds checking on variable assignments, such an assignment may cause the larger value to overflow into memory adjacent to the variable. Thus, when the value assigned to the variable is larger than the space allocated to the variable, a buffer overflow results. Malware programs can use the buffer overflow to take control of the memory adjacent to the variable, e.g., the call stack.
Another example of a vulnerability is a dangling pointer. A dangling pointer is a pointer that is created when the object of the pointer is deleted, but the value of the pointer is not changed. The pointer continues to point to a memory location that is no longer being used to store the object of the pointer. If, for example, the pointer was used to make a virtual function call, a malware program can exploit the dangling pointer to call a function at a different address. Thus, the malware program can use the dangling pointer to execute malicious code. Malware programs target other types of vulnerabilities also. Other examples of vulnerabilities that can be monitored are null pointer references, broken pointer references, integer overflows, etc. Vulnerabilities can also be exploited through scripts, such as JavaScript™. For example, malware programs typically encode malicious code in a call to the JavaScript™ “unescape” function.
These vulnerabilities can exist in different places in binary executables, including in both exported and nonexported functions. Exported functions in a binary executable are functions that are accessible by external references. For example, a DLL external to a binary executable can call an exported function within the binary executable through the external reference of the exported function. In another example, an exported function in a library can be called from a binary executable that is external to the library.
If there is a vulnerability in an exported function, a malware protection program can monitor any usage of the exported function by hooking function calls to the exported function. A malware protection program can hook function calls to the exported function by intercepting the function calls and monitoring the properties of the function calls to ensure that the function calls are not used to exploit a vulnerability. For example, if the malware protection program intercepts function calls to an exported function, it can monitor the values that are passed into the exported function and ensure that none of the values cause a buffer overflow in the exported function. Additionally, it can ensure that none of the values attempt to exploit a dangling pointer.
In contrast, nonexported functions are functions that are not designed to be called from external references. Thus, an external binary DLL cannot call a nonexported function in a binary executable because the binary executable does not maintain the location of nonexported functions within the binary executable. Nonexported functions are sometimes referred to as local functions, because they are only accessible from the binary executable in which the nonexported function resides. Vulnerabilities also exist in nonexported functions. If the vulnerability exists in a nonexported function, the malware protection program cannot hook the calls to the nonexported function by intercepting external function calls because there are no external references to a nonexported function. The only references to the nonexported function are internal to the binary file. The malware protection program can hook calls to exported functions that call the nonexported function with the vulnerability, but this is not an effective method of monitoring vulnerabilities in the nonexported function and it often results in false positives. This is because a malware protection program uses a hook to examine arguments that are passed into a function. If the vulnerability does not exist within the function that is hooked (i.e., the exported function), but rather, exists in a function that is referenced by the hooked function (i.e., the nonexported function), the malware protection program can only attempt to predict what arguments are used to exploit a vulnerability.